Dynamic Monitoring of High-Performance Distributed Applications

Developers and users of high-performance distributed systems often observe performance problems such as unexpectedly low throughput or high latency. Determining the source of the performance problems requires detailed end-to-end instrumentation of all components, including the applications, operating systems, hosts, and networks. However, one must be very careful to design the instru-mentation to have extremely low overhead, and not affect the system being monitored. In this paper we present a very lightweight instrumentation system that can be dynamically activated to unobtrusively collect and aggregate detailed end-to-end monitoring information from distributed applications. We also show how emerging " Web Services " can be used to facilitate remote interaction with this system. Developers and users of high-performance distributed systems often see unexpected performance problems. It can be difficult to track down the cause of these performance problems because distributed system components interact in complex ways. Bottlenecks can occur in any of the components through which the data flows: the applications, the operating systems, the device drivers, the network interfaces, and/or in network hardware such as switches and routers. In previous work we have shown that detailed application monitoring is vital for both performance analysis and application debugging [34][4][33]. In general we have found that performance analysis of distributed systems requires monitoring events before and after every I/O operation. This can generate huge amounts of monitoring data, and great care must be taken to deal with this data in an efficient and unobtrusive manner. In large cross-domain systems such as computational or data Grids, fine-grained mechanisms for dynamic control of the monitoring are also essential. Consider the use-case of monitoring some of the High Energy Physics (HEP) Grid projects [25][3][12] in a Data Grid environment. These projects, which will handle hundreds of terabytes of data, require detailed instrumentation data to understand and optimize their data transfers. For example, the user of a Grid File Replication service [5][38] notices that generating new replicas is taking much longer than it did last week. The user has no idea why performance has changed. Is there a problem in the network, disk, end host, GridFTP server, GridFTP client, or some other Grid middleware such as the authentication or authorization system? Monitoring information is needed to pinpoint the bottleneck, and determine what changed to cause this bottleneck. Current performance must be analyzed, and compared against a baseline drawn from previously archived information. This performance analysis requires monitoring data for …